{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing implementations of LibFM\n",
    "\n",
    "By **LibFM** I mean an approach to solve classification and regression problems.\n",
    "This approach is frequently used in recommendation systems, because it generalizes the matrix decompositions. \n",
    "\n",
    "**LibFM** proved to be quite useful to deal with highly categorical data (user id / movie id / movie language / site id / advertisement id / etc.).\n",
    "\n",
    "Implementations tested\n",
    "\n",
    "\n",
    "* Original and widely known implementation was written by Steffen Rendle (and available on [github](https://github.com/srendle/libfm)).\n",
    "    * contains SGD, SGDA, ALS and MCMC optimizers\n",
    "    * command-line interface, does not have official python / R wrapper\n",
    "    * does not provide a way to save / load trained formula. Each time you want to predict something, you need to restart training process \n",
    "    * has some extensions (that almost nobody uses)\n",
    "    * supports linux, mac os\n",
    "    * has non-oficial [pythonic wrapper](https://github.com/jfloff/pywFM)\n",
    "\n",
    "\n",
    "* FastFM ([github repo](https://github.com/ibayer/fastFM))\n",
    "    * claimed to be faster in the author's article\n",
    "    * has both command-line interface and convenient python wrapper, which *almost* follows scikit-learn conventions.\n",
    "    * supports SGD, ALS and MCMC optimizers\n",
    "    * supports save / load (for the except of MCMC)\n",
    "    * supports linux, mac os (though some issues with mac os)\n",
    "    \n",
    "    \n",
    "* pylibFM ([github repo](https://github.com/coreylynch/pyFM))\n",
    "    * uses SGDA\n",
    "    * pythonic library implemented with cython\n",
    "    * save / load operates normally\n",
    "    * supports any platform, provided cython operates normally\n",
    "    * slow and requires additional tuning, the number of iterations is reduced for pylibFM in tests\n",
    "    \n",
    "None of the libraries are pip-installable and all libraries need some manual setup. FastFM is the only to install itself normally into site-packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is tested\n",
    "\n",
    "ALS (alternating least squares) is very useful optimization technique for factorization models, however\n",
    "there is still one parameter one has to pass - namely, regularization. Quality of classification / regression is quite sensible to this parameter, so for fast tests data analyst prefers to leave the question of selecting regularization to machine learning.\n",
    "\n",
    "MCMC is usually proposed as a solution: optimization algorithm should \"find\" the optimal regularization. \n",
    "MCMC uses however some priors (which don't influence the result that much).\n",
    "\n",
    "So I am testing the quality libraries provide **without additional tuning** to check how bayesian inference and other heuristics work.\n",
    "\n",
    "\n",
    "## Logistic regression\n",
    "\n",
    "Logistic regression is used as a stable **baseline**, because it is basic method to work with highly categorical data.\n",
    "\n",
    "However, logistic regression, for instance, does not encounter the relation between user variables and movie variables (in the context of movie recommendations), so this approach is not able to provide any senseful recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "import pandas\n",
    "import load_problems\n",
    "import cPickle as pickle\n",
    "from sklearn.metrics import roc_auc_score, mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from fastFM.mcmc import FMClassification, FMRegression\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.linear_model import LogisticRegression, Ridge\n",
    "from sklearn.datasets import dump_svmlight_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining functions for benchmarking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "LIBFM_PATH = '/moosefs/ipython_env/python_libfm/bin/libFM'\n",
    "PYLIBFM_PATH = '/moosefs/ipython_env/python_pylibFM/'\n",
    "\n",
    "import sys\n",
    "if PYLIBFM_PATH not in sys.path:\n",
    "    sys.path.insert(0, PYLIBFM_PATH)\n",
    "import pylibfm\n",
    "\n",
    "\n",
    "def fitpredict_logistic(trainX, trainY, testX, classification=True, **params):\n",
    "    encoder = OneHotEncoder(handle_unknown='ignore').fit(trainX)\n",
    "    trainX = encoder.transform(trainX)\n",
    "    testX = encoder.transform(testX)\n",
    "    if classification:\n",
    "        clf = LogisticRegression(**params)\n",
    "        clf.fit(trainX, trainY)\n",
    "        return clf.predict_proba(testX)[:, 1]\n",
    "    else:\n",
    "        clf = Ridge(**params)\n",
    "        clf.fit(trainX, trainY)\n",
    "        return clf.predict(testX)\n",
    "\n",
    "def fitpredict_fastfm(trainX, trainY, testX, classification=True, rank=8, n_iter=100):\n",
    "    encoder = OneHotEncoder(handle_unknown='ignore').fit(trainX)\n",
    "    trainX = encoder.transform(trainX)\n",
    "    testX = encoder.transform(testX)\n",
    "    if classification:\n",
    "        clf = FMClassification(rank=rank, n_iter=n_iter)\n",
    "        return clf.fit_predict_proba(trainX, trainY, testX)\n",
    "    else:\n",
    "        clf = FMRegression(rank=rank, n_iter=n_iter)\n",
    "        return clf.fit_predict(trainX, trainY, testX)  \n",
    "\n",
    "def fitpredict_libfm(trainX, trainY, testX, classification=True, rank=8, n_iter=100):\n",
    "    encoder = OneHotEncoder(handle_unknown='ignore').fit(trainX)\n",
    "    trainX = encoder.transform(trainX)\n",
    "    testX = encoder.transform(testX)\n",
    "    train_file = 'libfm_train.txt'\n",
    "    test_file = 'libfm_test.txt'\n",
    "    with open(train_file, 'w') as f:\n",
    "        dump_svmlight_file(trainX, trainY, f=f)\n",
    "    with open(test_file, 'w') as f:\n",
    "        dump_svmlight_file(testX, numpy.zeros(testX.shape[0]), f=f)\n",
    "    task = 'c' if classification else 'r'\n",
    "    console_output = !$LIBFM_PATH -task $task -method mcmc -train $train_file -test $test_file -iter $n_iter -dim '1,1,$rank' -out output.libfm\n",
    "    \n",
    "    libfm_pred = pandas.read_csv('output.libfm', header=None).values.flatten()\n",
    "    return libfm_pred\n",
    "\n",
    "def fitpredict_pylibfm(trainX, trainY, testX, classification=True, rank=8, n_iter=10):\n",
    "    encoder = OneHotEncoder(handle_unknown='ignore').fit(trainX)\n",
    "    trainX = encoder.transform(trainX)\n",
    "    testX = encoder.transform(testX)\n",
    "    task = 'classification' if classification else 'regression'\n",
    "    fm = pylibfm.FM(num_factors=rank, num_iter=n_iter, verbose=False, task=task)\n",
    "    if classification:\n",
    "        fm.fit(trainX, trainY)\n",
    "    else:\n",
    "        fm.fit(trainX, trainY * 1.)\n",
    "    return fm.predict(testX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Executing all of the tests takes much time\n",
    "\n",
    "Below is simple mechanism, which preserves results between runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import OrderedDict\n",
    "import time\n",
    "\n",
    "all_results = OrderedDict()\n",
    "try:\n",
    "    with open('./saved_results.pkl') as f:\n",
    "        all_results = pickle.load(f)\n",
    "except:\n",
    "    pass\n",
    "\n",
    "def test_on_dataset(trainX, testX, trainY, testY, task_name, classification=True, use_pylibfm=True):\n",
    "    algorithms = OrderedDict()\n",
    "    algorithms['logistic'] = fitpredict_logistic\n",
    "    algorithms['libFM']    = fitpredict_libfm\n",
    "    algorithms['fastFM']   = fitpredict_fastfm\n",
    "    if use_pylibfm:\n",
    "        algorithms['pylibfm']  = fitpredict_pylibfm\n",
    "    \n",
    "    results = pandas.DataFrame()\n",
    "    for name, fit_predict in algorithms.items():\n",
    "        start = time.time()\n",
    "        predictions = fit_predict(trainX, trainY, testX, classification=classification)\n",
    "        spent_time = time.time() - start\n",
    "        results.ix[name, 'time'] = spent_time\n",
    "        if classification:\n",
    "            results.ix[name, 'ROC AUC'] = roc_auc_score(testY, predictions)\n",
    "        else:\n",
    "            results.ix[name, 'RMSE'] = numpy.mean((testY - predictions) ** 2) ** 0.5\n",
    "            \n",
    "    all_results[task_name] = results\n",
    "    with open('saved_results.pkl', 'w') as f:\n",
    "        pickle.dump(all_results, f)\n",
    "        \n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing on movielens-100k dataset, only ids\n",
    "\n",
    "MovieLens dataset is famous dataset in recommender systems. The task is to predict ratings for movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>movie</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>98980</th>\n",
       "      <td>810</td>\n",
       "      <td>900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69824</th>\n",
       "      <td>803</td>\n",
       "      <td>754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9928</th>\n",
       "      <td>51</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75599</th>\n",
       "      <td>734</td>\n",
       "      <td>180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95621</th>\n",
       "      <td>896</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  movie\n",
       "98980   810    900\n",
       "69824   803    754\n",
       "9928     51    286\n",
       "75599   734    180\n",
       "95621   896     95"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_movielens_100k(all_features=False)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>0.059469</td>\n",
       "      <td>0.942771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>8.970990</td>\n",
       "      <td>0.913520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>4.840041</td>\n",
       "      <td>0.915184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pylibfm</th>\n",
       "      <td>13.157164</td>\n",
       "      <td>0.944870</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               time      RMSE\n",
       "logistic   0.059469  0.942771\n",
       "libFM      8.970990  0.913520\n",
       "fastFM     4.840041  0.915184\n",
       "pylibfm   13.157164  0.944870"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, task_name='ml100k, ids', classification=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing on movielens-100k dataset, with additional information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>movie</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>released</th>\n",
       "      <th>unknown</th>\n",
       "      <th>Action</th>\n",
       "      <th>Adventure</th>\n",
       "      <th>...</th>\n",
       "      <th>Fantasy</th>\n",
       "      <th>Film-Noir</th>\n",
       "      <th>Horror</th>\n",
       "      <th>Musical</th>\n",
       "      <th>Mystery</th>\n",
       "      <th>Romance</th>\n",
       "      <th>Sci-Fi</th>\n",
       "      <th>Thriller</th>\n",
       "      <th>War</th>\n",
       "      <th>Western</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>98980</th>\n",
       "      <td>692</td>\n",
       "      <td>1310</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>615</td>\n",
       "      <td>68</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69824</th>\n",
       "      <td>931</td>\n",
       "      <td>528</td>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>59</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9928</th>\n",
       "      <td>216</td>\n",
       "      <td>553</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>110</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75599</th>\n",
       "      <td>798</td>\n",
       "      <td>498</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>166</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95621</th>\n",
       "      <td>910</td>\n",
       "      <td>547</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>397</td>\n",
       "      <td>68</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  movie  age  gender  occupation  zip  released  unknown  Action  \\\n",
       "98980   692   1310   33       0           7  615        68        0       0   \n",
       "69824   931    528   48       1           3   59        57        0       0   \n",
       "9928    216    553   12       1          13  110        67        0       1   \n",
       "75599   798    498   39       0           0  166        30        0       0   \n",
       "95621   910    547   27       0          20  397        68        0       0   \n",
       "\n",
       "       Adventure   ...     Fantasy  Film-Noir  Horror  Musical  Mystery  \\\n",
       "98980          0   ...           0          0       0        0        0   \n",
       "69824          0   ...           0          0       0        0        0   \n",
       "9928           1   ...           0          0       0        0        0   \n",
       "75599          0   ...           0          0       0        0        0   \n",
       "95621          0   ...           1          0       0        0        0   \n",
       "\n",
       "       Romance  Sci-Fi  Thriller  War  Western  \n",
       "98980        0       0         0    0        0  \n",
       "69824        0       0         0    0        0  \n",
       "9928         0       0         0    0        0  \n",
       "75599        0       0         0    0        0  \n",
       "95621        0       0         0    0        0  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_movielens_100k(all_features=True)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>1.869114</td>\n",
       "      <td>0.942377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>49.632649</td>\n",
       "      <td>0.896349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>53.611804</td>\n",
       "      <td>0.896543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pylibfm</th>\n",
       "      <td>55.756278</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               time      RMSE\n",
       "logistic   1.869114  0.942377\n",
       "libFM     49.632649  0.896349\n",
       "fastFM    53.611804  0.896543\n",
       "pylibfm   55.756278       NaN"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, task_name='ml100k', classification=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing on movielens-1m dataset, only ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "load_problems.py:73: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators; you can avoid this warning by specifying engine='python'.\n",
      "  names=['user', 'movie', 'rating', 'timestamp'], header=None)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>movie</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>610738</th>\n",
       "      <td>3703</td>\n",
       "      <td>3541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>324752</th>\n",
       "      <td>1923</td>\n",
       "      <td>756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>808217</th>\n",
       "      <td>4836</td>\n",
       "      <td>1288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133807</th>\n",
       "      <td>866</td>\n",
       "      <td>1106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>431857</th>\n",
       "      <td>2630</td>\n",
       "      <td>2857</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user  movie\n",
       "610738  3703   3541\n",
       "324752  1923    756\n",
       "808217  4836   1288\n",
       "133807   866   1106\n",
       "431857  2630   2857"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_movielens_1m(all_features=False)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>1.111601</td>\n",
       "      <td>0.910718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>275.672684</td>\n",
       "      <td>0.861539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>307.400295</td>\n",
       "      <td>0.858305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pylibfm</th>\n",
       "      <td>132.618739</td>\n",
       "      <td>0.870263</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                time      RMSE\n",
       "logistic    1.111601  0.910718\n",
       "libFM     275.672684  0.861539\n",
       "fastFM    307.400295  0.858305\n",
       "pylibfm   132.618739  0.870263"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, task_name='ml-1m,ids', classification=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing on movielens-1m dataset, with additional information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "load_problems.py:78: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators; you can avoid this warning by specifying engine='python'.\n",
      "  names=['user', 'gender', 'age', 'occupation', 'zip'], header=None)\n",
      "load_problems.py:79: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators; you can avoid this warning by specifying engine='python'.\n",
      "  movies = pandas.read_csv(folder + '/movies.dat', sep='::', names=['movie', 'title', 'genres'], header=None)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>movie</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>...</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>610738</th>\n",
       "      <td>5245</td>\n",
       "      <td>2240</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2168</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>324752</th>\n",
       "      <td>4260</td>\n",
       "      <td>2213</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>808217</th>\n",
       "      <td>481</td>\n",
       "      <td>1205</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>1878</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133807</th>\n",
       "      <td>973</td>\n",
       "      <td>1108</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "      <td>3140</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>431857</th>\n",
       "      <td>1344</td>\n",
       "      <td>2302</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2336</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        user  movie  gender  age  occupation   zip  0  1  2  3 ...  10  11  \\\n",
       "610738  5245   2240       0    1           0  2168  0  0  0  0 ...   0   0   \n",
       "324752  4260   2213       0    3           6   346  1  1  0  0 ...   0   0   \n",
       "808217   481   1205       1    2          14  1878  0  0  0  0 ...   0   0   \n",
       "133807   973   1108       1    3          19  3140  1  1  0  0 ...   0   0   \n",
       "431857  1344   2302       0    2           2  2336  0  0  0  0 ...   0   0   \n",
       "\n",
       "        12  13  14  15  16  17  18  19  \n",
       "610738   0   0   0   0   0   0   0   0  \n",
       "324752   0   0   0   1   0   0   0   0  \n",
       "808217   0   0   0   0   0   0   0   0  \n",
       "133807   0   0   0   0   0   0   0   0  \n",
       "431857   0   0   0   1   0   0   0   0  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_movielens_1m(all_features=True)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>23.983249</td>\n",
       "      <td>0.911024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>779.900802</td>\n",
       "      <td>0.850382</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>1170.468130</td>\n",
       "      <td>0.852738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pylibfm</th>\n",
       "      <td>564.922632</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 time      RMSE\n",
       "logistic    23.983249  0.911024\n",
       "libFM      779.900802  0.850382\n",
       "fastFM    1170.468130  0.852738\n",
       "pylibfm    564.922632       NaN"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, task_name='ml-1m', classification=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test on flights dataset - 1m\n",
    "\n",
    "Flights dataset is quite famous due to [these benchmarks](github.com/szilard/benchm-ml) by szilard. \n",
    "\n",
    "Based on defferent charateristics the goal is to predict whether the flight was delayed by 15 minutes or more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>DepTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c-4</td>\n",
       "      <td>c-26</td>\n",
       "      <td>c-2</td>\n",
       "      <td>1828</td>\n",
       "      <td>XE</td>\n",
       "      <td>LEX</td>\n",
       "      <td>IAH</td>\n",
       "      <td>828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>c-12</td>\n",
       "      <td>c-11</td>\n",
       "      <td>c-1</td>\n",
       "      <td>1212</td>\n",
       "      <td>UA</td>\n",
       "      <td>DEN</td>\n",
       "      <td>MCI</td>\n",
       "      <td>533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c-10</td>\n",
       "      <td>c-1</td>\n",
       "      <td>c-6</td>\n",
       "      <td>935</td>\n",
       "      <td>OH</td>\n",
       "      <td>HSV</td>\n",
       "      <td>CVG</td>\n",
       "      <td>325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c-11</td>\n",
       "      <td>c-26</td>\n",
       "      <td>c-6</td>\n",
       "      <td>930</td>\n",
       "      <td>OH</td>\n",
       "      <td>JFK</td>\n",
       "      <td>PNS</td>\n",
       "      <td>1028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c-12</td>\n",
       "      <td>c-6</td>\n",
       "      <td>c-2</td>\n",
       "      <td>1350</td>\n",
       "      <td>MQ</td>\n",
       "      <td>DFW</td>\n",
       "      <td>LBB</td>\n",
       "      <td>282</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Month DayofMonth DayOfWeek  DepTime UniqueCarrier Origin Dest  Distance\n",
       "0   c-4       c-26       c-2     1828            XE    LEX  IAH       828\n",
       "1  c-12       c-11       c-1     1212            UA    DEN  MCI       533\n",
       "2  c-10        c-1       c-6      935            OH    HSV  CVG       325\n",
       "3  c-11       c-26       c-6      930            OH    JFK  PNS      1028\n",
       "4  c-12        c-6       c-2     1350            MQ    DFW  LBB       282"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_flight(large=False, convert_to_ints=False)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>DepTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>19</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>21</td>\n",
       "      <td>157</td>\n",
       "      <td>133</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>79</td>\n",
       "      <td>172</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>129</td>\n",
       "      <td>72</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>147</td>\n",
       "      <td>220</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>13</td>\n",
       "      <td>80</td>\n",
       "      <td>155</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Month  DayofMonth  DayOfWeek  DepTime  UniqueCarrier  Origin  Dest  \\\n",
       "0      7          19          2        8             21     157   133   \n",
       "1      4           3          1        4             18      79   172   \n",
       "2      2           1          6        2             15     129    72   \n",
       "3      3          19          6        2             15     147   220   \n",
       "4      4          28          2        5             13      80   155   \n",
       "\n",
       "   Distance  \n",
       "0         6  \n",
       "1         4  \n",
       "2         2  \n",
       "3         7  \n",
       "4         2  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_flight(large=False, convert_to_ints=True)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>26.408147</td>\n",
       "      <td>0.715099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>441.534214</td>\n",
       "      <td>0.720484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>667.197644</td>\n",
       "      <td>0.718840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pylibfm</th>\n",
       "      <td>316.149734</td>\n",
       "      <td>0.711200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                time   ROC AUC\n",
       "logistic   26.408147  0.715099\n",
       "libFM     441.534214  0.720484\n",
       "fastFM    667.197644  0.718840\n",
       "pylibfm   316.149734  0.711200"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, task_name='flight1m', classification=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test on flights dataset - 10m\n",
    "\n",
    "pylibFM drops the kernel, so doesn't participate in comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>DepTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>37</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>149</td>\n",
       "      <td>157</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>29</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11</td>\n",
       "      <td>27</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>80</td>\n",
       "      <td>198</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7</td>\n",
       "      <td>19</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>216</td>\n",
       "      <td>155</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Month  DayofMonth  DayOfWeek  DepTime  UniqueCarrier  Origin  Dest  \\\n",
       "0      8           3          4        0             15      37    29   \n",
       "1      6          16          3        8              1     149   157   \n",
       "2      3          13          7        0             13      29    37   \n",
       "3     11          27          6        4             14      80   198   \n",
       "4      7          19          2        1             12     216   155   \n",
       "\n",
       "   Distance  \n",
       "0         1  \n",
       "1         9  \n",
       "2         1  \n",
       "3         7  \n",
       "4         9  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_flight(large=True, convert_to_ints=True)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>307.085924</td>\n",
       "      <td>0.715754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>8500.038059</td>\n",
       "      <td>0.724258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>10718.261802</td>\n",
       "      <td>0.721615</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  time   ROC AUC\n",
       "logistic    307.085924  0.715754\n",
       "libFM      8500.038059  0.724258\n",
       "fastFM    10718.261802  0.721615"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, task_name='flight10m', classification=True, use_pylibfm=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flights dataset with additional features\n",
    "\n",
    "We simply add some 'quadratic' features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Month</th>\n",
       "      <th>DayofMonth</th>\n",
       "      <th>DayOfWeek</th>\n",
       "      <th>DepTime</th>\n",
       "      <th>UniqueCarrier</th>\n",
       "      <th>Origin</th>\n",
       "      <th>Dest</th>\n",
       "      <th>Distance</th>\n",
       "      <th>UniqueCarrier_Origin</th>\n",
       "      <th>UniqueCarrier_Dest</th>\n",
       "      <th>UniqueCarrier_DepTime</th>\n",
       "      <th>Origin_Dest</th>\n",
       "      <th>Origin_DepTime</th>\n",
       "      <th>Dest_DepTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>19</td>\n",
       "      <td>2</td>\n",
       "      <td>19</td>\n",
       "      <td>21</td>\n",
       "      <td>157</td>\n",
       "      <td>133</td>\n",
       "      <td>7</td>\n",
       "      <td>1628</td>\n",
       "      <td>1622</td>\n",
       "      <td>496</td>\n",
       "      <td>2606</td>\n",
       "      <td>2612</td>\n",
       "      <td>2275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>18</td>\n",
       "      <td>79</td>\n",
       "      <td>172</td>\n",
       "      <td>5</td>\n",
       "      <td>1342</td>\n",
       "      <td>1369</td>\n",
       "      <td>417</td>\n",
       "      <td>1267</td>\n",
       "      <td>1294</td>\n",
       "      <td>2921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "      <td>129</td>\n",
       "      <td>72</td>\n",
       "      <td>3</td>\n",
       "      <td>1072</td>\n",
       "      <td>1050</td>\n",
       "      <td>344</td>\n",
       "      <td>1985</td>\n",
       "      <td>2117</td>\n",
       "      <td>1195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "      <td>147</td>\n",
       "      <td>220</td>\n",
       "      <td>8</td>\n",
       "      <td>1083</td>\n",
       "      <td>1111</td>\n",
       "      <td>344</td>\n",
       "      <td>2363</td>\n",
       "      <td>2410</td>\n",
       "      <td>3807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>13</td>\n",
       "      <td>80</td>\n",
       "      <td>155</td>\n",
       "      <td>3</td>\n",
       "      <td>806</td>\n",
       "      <td>836</td>\n",
       "      <td>299</td>\n",
       "      <td>1379</td>\n",
       "      <td>1317</td>\n",
       "      <td>2665</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Month  DayofMonth  DayOfWeek  DepTime  UniqueCarrier  Origin  Dest  \\\n",
       "0      7          19          2       19             21     157   133   \n",
       "1      4           3          1       13             18      79   172   \n",
       "2      2           1          6       10             15     129    72   \n",
       "3      3          19          6       10             15     147   220   \n",
       "4      4          28          2       14             13      80   155   \n",
       "\n",
       "   Distance  UniqueCarrier_Origin  UniqueCarrier_Dest  UniqueCarrier_DepTime  \\\n",
       "0         7                  1628                1622                    496   \n",
       "1         5                  1342                1369                    417   \n",
       "2         3                  1072                1050                    344   \n",
       "3         8                  1083                1111                    344   \n",
       "4         3                   806                 836                    299   \n",
       "\n",
       "   Origin_Dest  Origin_DepTime  Dest_DepTime  \n",
       "0         2606            2612          2275  \n",
       "1         1267            1294          2921  \n",
       "2         1985            2117          1195  \n",
       "3         2363            2410          3807  \n",
       "4         1379            1317          2665  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_flight_extended(large=False)\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>130.553217</td>\n",
       "      <td>0.739113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>1284.598730</td>\n",
       "      <td>0.760524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>2398.107206</td>\n",
       "      <td>0.748448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pylibfm</th>\n",
       "      <td>514.768003</td>\n",
       "      <td>0.743312</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 time   ROC AUC\n",
       "logistic   130.553217  0.739113\n",
       "libFM     1284.598730  0.760524\n",
       "fastFM    2398.107206  0.748448\n",
       "pylibfm    514.768003  0.743312"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, task_name='flight1m, ext', classification=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test on Avazu dataset (100k)\n",
    "\n",
    "Avazu dataset comes from kaggle challenge, goal is to predict Click-Through Rate. \n",
    "\n",
    "All the variables given are categorical, LibFM gave good results in this challenge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hour</th>\n",
       "      <th>C1</th>\n",
       "      <th>banner_pos</th>\n",
       "      <th>site_id</th>\n",
       "      <th>site_domain</th>\n",
       "      <th>site_category</th>\n",
       "      <th>app_id</th>\n",
       "      <th>app_domain</th>\n",
       "      <th>app_category</th>\n",
       "      <th>device_id</th>\n",
       "      <th>...</th>\n",
       "      <th>device_conn_type</th>\n",
       "      <th>C14</th>\n",
       "      <th>C15</th>\n",
       "      <th>C16</th>\n",
       "      <th>C17</th>\n",
       "      <th>C18</th>\n",
       "      <th>C19</th>\n",
       "      <th>C20</th>\n",
       "      <th>C21</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13458308</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>582</td>\n",
       "      <td>339</td>\n",
       "      <td>2</td>\n",
       "      <td>884</td>\n",
       "      <td>254</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>281</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11331426</th>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>582</td>\n",
       "      <td>339</td>\n",
       "      <td>2</td>\n",
       "      <td>884</td>\n",
       "      <td>254</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>283</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1271792</th>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>879</td>\n",
       "      <td>903</td>\n",
       "      <td>24</td>\n",
       "      <td>884</td>\n",
       "      <td>254</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>84</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "      <td>14</td>\n",
       "      <td>57</td>\n",
       "      <td>6</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7618971</th>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>762</td>\n",
       "      <td>841</td>\n",
       "      <td>24</td>\n",
       "      <td>884</td>\n",
       "      <td>254</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>88</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>220</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>132</td>\n",
       "      <td>42</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16882663</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>158</td>\n",
       "      <td>893</td>\n",
       "      <td>24</td>\n",
       "      <td>884</td>\n",
       "      <td>254</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>376</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>88</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "      <td>8</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          hour  C1  banner_pos  site_id  site_domain  site_category  app_id  \\\n",
       "13458308     1   2           0      582          339              2     884   \n",
       "11331426    10   2           0      582          339              2     884   \n",
       "1271792      6   2           0      879          903             24     884   \n",
       "7618971     13   2           0      762          841             24     884   \n",
       "16882663     3   2           1      158          893             24     884   \n",
       "\n",
       "          app_domain  app_category  device_id ...   device_conn_type  C14  \\\n",
       "13458308         254             0        800 ...                  0  281   \n",
       "11331426         254             0        800 ...                  1  283   \n",
       "1271792          254             0        800 ...                  0   84   \n",
       "7618971          254             0        800 ...                  0   88   \n",
       "16882663         254             0        800 ...                  0  376   \n",
       "\n",
       "          C15  C16  C17  C18  C19  C20  C21  day  \n",
       "13458308    3    2   56    0    2    0   22   24  \n",
       "11331426    3    2   56    0    2    0   22   23  \n",
       "1271792     3    2   18    3   14   57    6   21  \n",
       "7618971     3    2  220    0    2  132   42   22  \n",
       "16882663    3    2   88    2    4   13    8   25  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_ad(train_size=100000)\n",
    "# taking max hash of 1000 for each category\n",
    "trainX = trainX % 1000\n",
    "testX = testX % 1000\n",
    "trainX.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>40.417285</td>\n",
       "      <td>0.717096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>7778.470057</td>\n",
       "      <td>0.730173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>6601.202590</td>\n",
       "      <td>0.699962</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 time   ROC AUC\n",
       "logistic    40.417285  0.717096\n",
       "libFM     7778.470057  0.730173\n",
       "fastFM    6601.202590  0.699962"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_on_dataset(trainX, testX, trainY, testY, \n",
    "                task_name='avazu100k', classification=True, use_pylibfm=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Avazu 1m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>228.501853</td>\n",
       "      <td>0.740987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>9109.968575</td>\n",
       "      <td>0.748516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>7962.800504</td>\n",
       "      <td>0.733875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 time   ROC AUC\n",
       "logistic   228.501853  0.740987\n",
       "libFM     9109.968575  0.748516\n",
       "fastFM    7962.800504  0.733875"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX, testX, trainY, testY = load_problems.load_problem_ad(train_size=1000000)\n",
    "# taking max hash of 1000 for each category\n",
    "trainX = trainX % 1000\n",
    "testX = testX % 1000\n",
    "test_on_dataset(trainX, testX, trainY, testY, \n",
    "                task_name='avazu1m', classification=True, use_pylibfm=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results\n",
    "\n",
    "composing all results in one table. \n",
    "RMSE should be minimal, ROC AUC - maximal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>dataset</th>\n",
       "      <th colspan=\"2\" halign=\"left\">ml100k, ids</th>\n",
       "      <th colspan=\"2\" halign=\"left\">ml-1m,ids</th>\n",
       "      <th colspan=\"2\" halign=\"left\">ml100k</th>\n",
       "      <th colspan=\"2\" halign=\"left\">ml-1m</th>\n",
       "      <th colspan=\"2\" halign=\"left\">flight1m</th>\n",
       "      <th colspan=\"2\" halign=\"left\">flight1m, ext</th>\n",
       "      <th colspan=\"2\" halign=\"left\">flight10m</th>\n",
       "      <th colspan=\"2\" halign=\"left\">avazu100k</th>\n",
       "      <th colspan=\"2\" halign=\"left\">avazu1m</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>value</th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>time</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>time</th>\n",
       "      <th>ROC AUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic</th>\n",
       "      <td>0.059469</td>\n",
       "      <td>0.942771</td>\n",
       "      <td>1.111601</td>\n",
       "      <td>0.910718</td>\n",
       "      <td>1.869114</td>\n",
       "      <td>0.942377</td>\n",
       "      <td>23.983249</td>\n",
       "      <td>0.911024</td>\n",
       "      <td>26.408147</td>\n",
       "      <td>0.715099</td>\n",
       "      <td>130.553217</td>\n",
       "      <td>0.739113</td>\n",
       "      <td>307.085924</td>\n",
       "      <td>0.715754</td>\n",
       "      <td>40.417285</td>\n",
       "      <td>0.717096</td>\n",
       "      <td>228.501853</td>\n",
       "      <td>0.740987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>libFM</th>\n",
       "      <td>8.970990</td>\n",
       "      <td>0.913520</td>\n",
       "      <td>275.672684</td>\n",
       "      <td>0.861539</td>\n",
       "      <td>49.632649</td>\n",
       "      <td>0.896349</td>\n",
       "      <td>779.900802</td>\n",
       "      <td>0.850382</td>\n",
       "      <td>441.534214</td>\n",
       "      <td>0.720484</td>\n",
       "      <td>1284.598730</td>\n",
       "      <td>0.760524</td>\n",
       "      <td>8500.038059</td>\n",
       "      <td>0.724258</td>\n",
       "      <td>7778.470057</td>\n",
       "      <td>0.730173</td>\n",
       "      <td>9109.968575</td>\n",
       "      <td>0.748516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fastFM</th>\n",
       "      <td>4.840041</td>\n",
       "      <td>0.915184</td>\n",
       "      <td>307.400295</td>\n",
       "      <td>0.858305</td>\n",
       "      <td>53.611804</td>\n",
       "      <td>0.896543</td>\n",
       "      <td>1170.468130</td>\n",
       "      <td>0.852738</td>\n",
       "      <td>667.197644</td>\n",
       "      <td>0.718840</td>\n",
       "      <td>2398.107206</td>\n",
       "      <td>0.748448</td>\n",
       "      <td>10718.261802</td>\n",
       "      <td>0.721615</td>\n",
       "      <td>6601.202590</td>\n",
       "      <td>0.699962</td>\n",
       "      <td>7962.800504</td>\n",
       "      <td>0.733875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pylibfm</th>\n",
       "      <td>13.157164</td>\n",
       "      <td>0.944870</td>\n",
       "      <td>132.618739</td>\n",
       "      <td>0.870263</td>\n",
       "      <td>55.756278</td>\n",
       "      <td>NaN</td>\n",
       "      <td>564.922632</td>\n",
       "      <td>NaN</td>\n",
       "      <td>316.149734</td>\n",
       "      <td>0.711200</td>\n",
       "      <td>514.768003</td>\n",
       "      <td>0.743312</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "dataset  ml100k, ids             ml-1m,ids               ml100k            \\\n",
       "value           time      RMSE        time      RMSE       time      RMSE   \n",
       "logistic    0.059469  0.942771    1.111601  0.910718   1.869114  0.942377   \n",
       "libFM       8.970990  0.913520  275.672684  0.861539  49.632649  0.896349   \n",
       "fastFM      4.840041  0.915184  307.400295  0.858305  53.611804  0.896543   \n",
       "pylibfm    13.157164  0.944870  132.618739  0.870263  55.756278       NaN   \n",
       "\n",
       "dataset         ml-1m              flight1m           flight1m, ext            \\\n",
       "value            time      RMSE        time   ROC AUC          time   ROC AUC   \n",
       "logistic    23.983249  0.911024   26.408147  0.715099    130.553217  0.739113   \n",
       "libFM      779.900802  0.850382  441.534214  0.720484   1284.598730  0.760524   \n",
       "fastFM    1170.468130  0.852738  667.197644  0.718840   2398.107206  0.748448   \n",
       "pylibfm    564.922632       NaN  316.149734  0.711200    514.768003  0.743312   \n",
       "\n",
       "dataset      flight10m              avazu100k                avazu1m            \n",
       "value             time   ROC AUC         time   ROC AUC         time   ROC AUC  \n",
       "logistic    307.085924  0.715754    40.417285  0.717096   228.501853  0.740987  \n",
       "libFM      8500.038059  0.724258  7778.470057  0.730173  9109.968575  0.748516  \n",
       "fastFM    10718.261802  0.721615  6601.202590  0.699962  7962.800504  0.733875  \n",
       "pylibfm            NaN       NaN          NaN       NaN          NaN       NaN  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_table = pandas.DataFrame()\n",
    "tuples = []\n",
    "\n",
    "for name in ['ml100k, ids', 'ml-1m,ids', 'ml100k', 'ml-1m', 'flight1m', 'flight1m, ext', 'flight10m', 'avazu100k', 'avazu1m']:\n",
    "    df = all_results[name]\n",
    "    results_table[name + ' (time)'] = df['time']\n",
    "    metric_name = df.columns[-1]\n",
    "    results_table[name + metric_name] = df[metric_name]\n",
    "    tuples.append([name, 'time'])\n",
    "    tuples.append([name, df.columns[-1]])\n",
    "    \n",
    "results_table = results_table.T\n",
    "results_table.index = pandas.MultiIndex.from_tuples(tuples, names=['dataset', 'value'])\n",
    "results_table.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "- `pylibfm` is out of the game. It is slow, it crashes on large datasets, sometimes simply diverge and hardly can compete in quality. <br />\n",
    "   Nothing new, adaptive methods require babysitting\n",
    "- `FastFM` and `LibFM` are quite stable and fast\n",
    "-  but `LibFM`, being a bit faster, on average provides much better results.\n",
    "\n",
    "As a sidenote, we saw on example with flight dataset that some feature engineering with providing quadratic features gives very significant boost in quality - even logisitic regression can work much better and faster than FMs on original features."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
